Draft: Arc runner computeenv #16750

maikenp · 2023-09-27T14:46:02Z

This PR is related to #16653

It is based on the same branch, but I am now using the tool_script.sh as the executable that ARC runs.
I also demonstrate a very brute-force and simplistic way of using the compute_environment to rewrite the input paths in order for ARC to get the correct paths for the command-line when running on the remote site (that does not have a shared directory with the galaxy server).

I am still using a custom ARC tool - to avoid complications with complex path resolutions like e.g. this tool: https://github.com/galaxyproject/tools-iuc/blob/main/tools/minimap2/minimap2.xml where there are softlinks: ln -f -s '/storage/galaxy/data/datasets/4/5/c/dataset_45cf1bcb-28b1-4167-878d-1fb17636064e.dat' reference.fa && minimap2 --q-occ-frac 0.01 -t ${GALAXY_SLOTS:-4} reference.fa '/storage/galaxy/data/datasets/4/5/c/dataset_45cf1bcb-28b1-4167-878d-1fb17636064e.dat' -a | samtools view --no-PG -hT reference.fa | samtools sort -@${GALAXY_SLOTS:-2} -T "${TMPDIR:-.}" -O BAM -o '/storage/galaxy/data/jobs/000/228/outputs/dataset_03e29ceb-8733-4836-8981-90391c58105f.dat' in the command-line created - and I do not currently know how to solve this.

The custom test prototyping tool is:

<tool id="hello_arc" name="ARC hello example">
  <description>is a simple hello world python script</description>

 <command detect_errors="exit_code"><![CDATA[  
 /bin/bash '$arcjob_exe' "$message" --test >> command_out.txt; 
 /bin/echo "Some post processing job"  >> command_out.txt
 ]]>
 </command>
  
  <inputs>
   <param name="message" type="text" value="Hi from galaxy job wrapper" label="Hello message"/>
   <param name="arcjob_exe" type="data" label="ARC executable"/>
   <param name="arcjob_remote_filelist" type="data"  label="Upload file with list of remote file source paths"/>
 </inputs>

 
 <outputs>
   <collection name="output_job" type="list"  label="ARC job outputs">
     <discover_datasets pattern="(?P&lt;designation&gt;.+.*\..*)\.*" directory="./"   recurse="true" />
   </collection>
   

   <collection name="output_all" type="list"  format="txt" label="ARC job logs">
     <discover_datasets pattern="(?P&lt;designation&gt;.+.*).*" directory="./gmlog/"   recurse="true" />
   </collection>

   <!-- This does not work as galaxy moves the data source to the jobs workdir in galaxy - and if I rewrite the path then galaxy does not find the correct path obv. 
	Not sure how to disentable the moving from the data folder to the jobs folder, and the path rewrite for the remote resource -->
   <!--<data name="outputfile" format="txt"/>-->
 </outputs>
 
 <help>
   A tool used for testing the basic features for a job running on a remote site with ARC. 
  </help>
</tool>

This tool demonstrates uploading of local galaxy files to the remote ARC server. Also downloading of all output from the job on the ARC side to galaxy (however note, the pattern for finding the ARC logs does not work currently, wrong pattern, the files are present in the galaxy job dir though).
In addition, the tool - just for testing purposes - expects a file with a list of remote input files that ARC on the remote server side will download there, not involving Galaxy at all. This is not the solution that we will end up with, but it is for demonstrating the datastaging capability in ARC.

The tool is tested using the following file-list file:

$ cat remote_list.txt
http://download.nordugrid.org/repos/6/centos/el7/source/updates/repodata/repomd.xml
http://download.nordugrid.org/repos/6/rocky/9/source/updates/repodata/repomd.xml

Also, for testing, upload a bash script that is called by tool_script.sh for example

#!/bin/bash

/bin/echo "Hello from runscript to arcout1.txt" >> ./arcout1.txt
/bin/echo "Running on compute node: $(hostname)" >> ./arcout1.txt
/bin/echo "Time is $(date)" >> ./arcout1.txt
/bin/echo "Sleeping for 60s " >> ./arcout1.txt

/bin/echo "Hello from runscript to arcout2.txt" >> ./arcout2.txt
/bin/echo "Running on compute node: $(hostname)" >> ./arcout2.txt
/bin/echo "Time is $(date)" >> ./arcout2.txt
/bin/echo "Sleeping for 60s " >> ./arcout2.txt

/bin/sleep 60

>&1 /bin/echo "Some output to stdout from the runhello.sh executable."
>&1 /bin/echo "More output to stdout from runhello.sh - this is from message tag in tool:  $1"

The tool_script.sh produced by Galaxy is:

/bin/bash './runhello.sh' "Hi from galaxy job wrapper" --test >> command_out.txt; /bin/echo "Some post processing job"  >> command_out.txt

Once the job is done the workdir of the galaxy job will look like this:

[root@galaxy-arc-test-fresh ~]# ls -lhrt /storage/galaxy/data/jobs/000/284
total 12K
drwxr-xr-x. 2 galaxy galaxy   6 Sep 27 15:03 inputs
-rwxr-xr-x. 1 galaxy galaxy 347 Sep 27 15:03 tool_script.sh
drwxr-xr-x. 3 galaxy galaxy  26 Sep 27 15:05 working
drwxr-xr-x. 2 galaxy galaxy  44 Sep 27 15:05 outputs
-rw-r--r--. 1 galaxy galaxy  59 Sep 27 15:05 galaxy_284.o
-rw-r--r--. 1 galaxy galaxy  59 Sep 27 15:05 galaxy_284.e
[root@galaxy-arc-test-fresh ~]# ls -lhrt /storage/galaxy/data/jobs/000/284/working/
total 0
drwxr-xr-x. 3 galaxy galaxy 151 Sep 27 15:05 2812507f8fca
[root@galaxy-arc-test-fresh ~]# ls -lhrt /storage/galaxy/data/jobs/000/284/working/2812507f8fca/
total 20K
drwxr-xr-x. 2 galaxy galaxy 154 Sep 27 15:05 gmlog
-rw-r--r--. 1 galaxy galaxy   0 Sep 27 15:05 arc.out
-rw-r--r--. 1 galaxy galaxy   0 Sep 27 15:05 arc.err
-rw-r--r--. 1 galaxy galaxy 160 Sep 27 15:05 arcout1.txt
-rw-r--r--. 1 galaxy galaxy 160 Sep 27 15:05 arcout2.txt
-rw-r--r--. 1 galaxy galaxy 183 Sep 27 15:05 command_out.txt
-rw-r--r--. 1 galaxy galaxy 648 Sep 27 15:05 runhello.sh
-rw-r--r--. 1 galaxy galaxy 347 Sep 27 15:05 tool_script.sh
[root@galaxy-arc-test-fresh ~]# ls -lhrt /storage/galaxy/data/jobs/000/284/working/2812507f8fca/gmlog/
total 176K
-rw-r--r--. 1 galaxy galaxy  781 Sep 27 15:05 local
-rw-r--r--. 1 galaxy galaxy 141K Sep 27 15:05 errors
-rw-r--r--. 1 galaxy galaxy 1.1K Sep 27 15:05 description
-rw-r--r--. 1 galaxy galaxy  317 Sep 27 15:05 diag
-rw-r--r--. 1 galaxy galaxy    8 Sep 27 15:05 status
-rw-r--r--. 1 galaxy galaxy 1.7K Sep 27 15:05 xml
-rw-r--r--. 1 galaxy galaxy    0 Sep 27 15:05 input
-rw-r--r--. 1 galaxy galaxy    2 Sep 27 15:05 output
-rw-r--r--. 1 galaxy galaxy   30 Sep 27 15:05 input_status
-rw-r--r--. 1 galaxy galaxy  367 Sep 27 15:05 statistics

That contains all the ARC logs, in addition to the jobs output files.

This version of the ARC runner depends on pyarcrest 0.3: https://pypi.org/search/?q=pyarcrest (temporarily uploaded for testing purposes, later it will be included in the nordugrid ARC distro http://www.nordugrid.org/arc/arc6/common/repos/repository.html)

How to test the changes?

(Select all options that apply)

I've included appropriate automated tests.
This is a refactoring of components with existing test coverage.
Instructions for manual testing are as follows - mostly same as PR Adding new ARC job runner #16653
- For the galaxy admin:
  - Install the ARC runner (in this PR) - the pyarcrest (ARC REST client) will be installed as a dependency.
  - Configure the galaxy site to use the WLCG oidc backend ( This PR has now been merged: Adding wlcg oidc endpoint. #16617)
  - Configure the galaxy site with the ARC test tool (see above)
  - Configure the galaxy site with the user_extra_preferences ARC distributed computing form: Added simple form for adding ARC compute element endpoint usegalaxy-eu/infrastructure-playbook#879
- For the user:
  - Once the site is configured as above, you need to get authorized to get a token from WLCG IAM: https://wlcg.cloud.cnaf.infn.it/login Then log into galaxy via the WLCG login-button.
  - Set your user-preferences to use an ARC endpoint that you have access to. You can use the test-ARC endpoint https://arctestcluster-slurm-el8-arc7-ce1.cern-test.uiocloud.no
  - Submit the job - and sit back and relax.

To progress making this closer to production ready

How to sort out the path-rewrites. In ARC all input and output files need to be tagged as such, and from Galaxy I can only identify the input and output files from `job_wrapper.get_job().get_input_datasets() and .get_output_datasets(). But handling reference data that lives on the galaxy server - if those files can not be retrieved from the job_wrapper then ARC will not work for those tools.
Understand what is necessary in Galaxy for using the inbuilt ARC datastaging capabilities to the fullest
- If given the URI of a file, ARC will download this file to the remote server before the job starts running on the servers compute nodes - if the file was already downloaded before (by the user or others) it will be in the cache and no download is necessary.
Discussion with Marius related to solving complex output resolutions in galaxy. A little snippet of that is here:

License

I agree to license these and all my past contributions to the core galaxy codebase under the MIT license.

Adding pyarcrest conditional requirement for the ArcRESTJobRunner

ArcRESTJobRunner

Require version 0.2 which is compatible with the ARC job runner.

…cb9cc4 by jmchilton.

…from jmchilton jmchilton@96d3807

…hilton@1906b2b

…cessary job_actions method. Further improvements to the action on jobs will come.

…n part.

…e. And rewriting input-paths.

enolfc · 2024-01-04T16:02:42Z

lib/galaxy/jobs/runners/arc.py

+            return
+        """ prepare_job() calls prepare() but not allowing to pass a compute_environment object
+        As I need to define my own compute_environment for the remote compute I must call it here passing the compute_environment
+        TODO - not a good solution"""


I'm facing a similar problem with the DIRAC jobrunner and thought of modifying the prepare_job to take an extra argument compute_environment (default None), so it should keep working with existing implementations but allow for customisation

volodymyrss · 2024-09-13T08:34:04Z

Thanks for working on this, we need it so much in our astroparticle physics domains!

maikenp and others added 25 commits September 1, 2023 14:12

Update conditional-requirements.txt

3c2c67f

Adding pyarcrest conditional requirement for the ArcRESTJobRunner

Update __init__.py

54770d1

ArcRESTJobRunner

explicitly document default of multiple

bffcef8

Merge remote-tracking branch 'upstream/dev' into dev

7b8f5b6

Adding new job runner - the ARC job runner

1dc557e

Merge branch 'pyarcrest-dep' into arc_job_runner

46a53a4

Update conditional-requirements.txt

729c275

Require version 0.2 which is compatible with the ARC job runner.

Reusable abstractions for dealing with pyarcrest. Applying jmchilton@6…

3a29632

…cb9cc4 by jmchilton.

Refactor XML parsing out of middle of arc job runner. Applying patch …

75c07fd

…from jmchilton jmchilton@96d3807

Remove unused imports in arc runner. Applied patch from jmchilton jmc…

63d5002

…hilton@1906b2b

Fixed one job description omission related to executable. Removed une…

c30b651

…cessary job_actions method. Further improvements to the action on jobs will come.

linting

2a07bbc

More linting

7c8f1fd

Refactored to remove need for internal job id mapping.

b6acefb

Refactoring and simplification. Especially for the ARC job descriptio…

91a20f3

…n part.

Fix last-minute change typo

88451b1

Forgot to also commit the changed arc_util.py

dda0a0e

Remove unnecessary open mode parameters

894cc9a

Remove comment - linting

bd1d30f

Hopefully fixed all linting errors now

678dd76

more linting of whitespace

83417f8

More linting and some bugfix related to external job id

75087af

linting whitespace

d1f2fb0

The ARC runner augmented to work with the tool_script.sh as executabl…

140ef15

…e. And rewriting input-paths.

Linting done

09ce2e2

github-actions bot added area/jobs area/dependencies labels Sep 27, 2023

github-actions bot added this to the 23.2 milestone Sep 27, 2023

maikenp marked this pull request as draft September 27, 2023 18:39

This version of the ARC job runner relies on pyarcrest 0.3

9acf29d

mvdbeek modified the milestones: 23.2, 24.0 Dec 19, 2023

enolfc reviewed Jan 4, 2024

View reviewed changes

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Draft: Arc runner computeenv #16750

Draft: Arc runner computeenv #16750

maikenp commented Sep 27, 2023 •

edited

Loading

enolfc Jan 4, 2024

volodymyrss commented Sep 13, 2024

Draft: Arc runner computeenv #16750

Are you sure you want to change the base?

Draft: Arc runner computeenv #16750

Conversation

maikenp commented Sep 27, 2023 • edited Loading

How to test the changes?

To progress making this closer to production ready

License

enolfc Jan 4, 2024

Choose a reason for hiding this comment

volodymyrss commented Sep 13, 2024

maikenp commented Sep 27, 2023 •

edited

Loading